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Optical pattern formation with a 2-level nonlinearity 
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We present an experimental and theoretical investigation of spontaneous pattern formation in the 
transverse section of a single retro-reflected laser beam passing through a cloud of cold Rubidium 
atoms. In contrast to previously investigated systems, the nonlinearity at work here is that of a 
2-level atom, which realizes the paradigmatic situation considered in many theoretical studies of 
optical pattern formation. In particular, we are able to observe the disappearance of the patterns 
at high intensity due to the intrinsic saturable character of 2-level atomic transitions. 
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Spontaneous pattern formation from a homogeneous 
state is a widespread phenomenon in nonlinear systems 
out of equilibrium [l|, . Originating from fields such as 


chemistry 


and hydrodynamics 


.g trom I 

ESi, 


the study of 


pattern formation has known a rapid development in op¬ 


tics starting from the 80’s 
such as a Kerr medium [^,1T 


. Paradigmatic examples 
or a collection of 2-level 
atoms at rest ll| were considered in early theoretical 
studies. Various nonlinear systems, either active such as 
lasers or photorefractive oscillators [1, [H, [I^ , or passive 
such as liquid crystals Bill , were used to realize the 
first experiments. In some range of experimental condi¬ 
tions, these nonlinear materials mimic ideal systems such 
as the Kerr medium but one usually lacks a complete the¬ 
oretical description of the light-matter interaction. Hot 
atomic vapors were also extensively employed to study 
various optical instabilities There, a theoretical 

description of the light-atom interaction is available, but 
the specific experimental conditions (Doppler broaden¬ 
ing, hyperfine structure, ballistic or diffusive motion of 
the atoms) considerably complicate the interpretation. 
Finally, cold atomic samples started recently to be em¬ 
ployed in optical pattern formation [2ll - E^ . 

We have identified in our single feedback mirror exper¬ 
iments three distinct mechanisms that lead to the spon¬ 
taneous formation of patterns. The first one, the op¬ 
tomechanical mechanism, is specific to cold atoms and 
relies on the spatial bunching of the atoms under the ac¬ 
tion of dipole forces. This mechanism is very efficient 
at ultracold temperature and leads to spectacular self¬ 
organization phenomena 2M ■ The presence of a Zeeman 
structure in the atomic ground state (spin degree of free¬ 
dom) allows for optical pumping, i.e. a redistribution 
between populations or creation of coherences between 
Zeeman substates, in particular within the ground-state. 
This mechanism is responsible for the polarization insta¬ 


e polarization msta- 


bilities studied in hot atomic vapors 
Finally, under specific experimental conditions the atoms 
behave as 2-level systems and the optical nonlinearity is 
only due to the saturation of the electric dipole tran¬ 
sition i.e population transfer between ground and ex¬ 



FIG. 1. (Color on line) Observation of patterns, (a) Experi¬ 
mental scheme, (b) Typical single-shot light distributions ob¬ 
served in the transverse instability regime, in the near (left) 
and far (right) field. The pump parameters are: I = 0.47 
W/cm^ and 5 = -b6.5 T. 


cited states. This is the situation studied in this pa¬ 
per, which realizes the paradigmatic theoretical descrip¬ 
tion of a homogeneously-broadened 2-level atomic tran¬ 
sition ll|. To our knowledge, the only previous experi¬ 


mental investigation of patterns due to a saturable non¬ 
linearity was achieved in a hot sodium vapor [l6| , and did 
not report the observation of the vanishing of the effect 
at large saturations. We present in this paper a detailed 
experimental investigation of this 2-level instability, and 
obtain a qualitative and quantitative agreement with a 
theoretical model based only on the microscopic descrip¬ 
tion of the atom-light interaction. 

The experimental setup, sketched in Fig.lHa), exploits 
the single-mirror feedback scheme (see [i^ for de¬ 
tails) . A Gaussian laser beam (referred to as “the pump” 
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in the following) of waist w = 1.47 mm and wavelength 
A = 780.2 nm is sent through a cold (T = 200 /xK) 
cloud of Rb®^ atoms, released from a large magneto¬ 
optical trap (MOT). The cloud has a typical size of 9 mm 
FWHM along the pump propagation axis and contains 
10^^ atoms. The resulting optical density (OD), for a 
weak beam on resonance with the F = 2 ^ F' = 3 tran¬ 
sition, is around 210. The linearly-polarized pump beam 
is retro-reflected by a mirror located at a distance d after 
the cloud (the vertical arrows in Fig. UKa) indicate the 
polarization of the beams). We use an imaging telescope 
(not represented in Fig.[IJa)) located between the MOT 
and the mirror to create a “virtual mirror”, which pro¬ 
vides an access to negative values of d The overall 
reflectivity of the feedback system is around 95% (cloud 
absorption not included). 

By selecting a short (duration < 1 /is) pump pulse, 
we can neglect the optomechanical nonlinearity which 
requires tens of /xs since the atoms have to move over 
distances of the order of « lOO/xm [ 2 ^. Other mech¬ 
anisms, relying on Zeeman internal degrees of freedom, 
also lead to a transverse instability and are currently un¬ 
der study in our groups. However, in the setup studied 
in the present paper a polarizer placed inside the feed¬ 
back loop (Fig. UKa)) guarantees that the feedback only 
occurs in the polarization channel parallel to the inci¬ 
dent pump polarization, and prevents the occurrence of a 
polarization instability linked to Zeeman degrees of free¬ 
dom. This is confirmed by the fact that almost no light 
is detected in the polarization channel orthogonal to the 
pump polarization. Throughout this work we are thus 
left with the simplest, 2-level nonlinearity corresponding 
to the following expression for the refractive index of our 
cloud of cold atoms: 

1 P in 

” 4^21 + {25/vy 1 + s ^ ^ 

where 5 = uii — ujo is the detuning between the laser 
and atomic frequencies, F = 27r x 6.06 MHz is the natu¬ 
ral width and p denotes the spatial atomic density. The 
nonlinearity arises from the presence in this expression 
of the saturation parameter s = 7 -^ i+ 4 (l/r)^ > '"^here 
I sat = 1-67 mW/cm^ is the saturation intensity. For 
s <C 1, the gas exhibits a Kerr-like behavior n ~ nQ+n 2 I, 
where ng is the linear refractive index and 7 x 2 the nonlin¬ 
ear one. Importantly, for s !§> 1 the nonlinearity vanishes 
and the instability is expected to disappear. In the Kerr 
regime, the nonlinearity is “self-focusing” ( 7 x 2 > 0) for 
(5 > 0 and “self-defocusing” for <5 < 0. Note that Eq. [T] 
only describes the real part of the complex refractive in¬ 
dex, which is responsible for the instability. However, 
(nonlinear) absorption is also present and included in our 
theoretical analysis. 

Fig. [Ijb) shows images of the transverse intensity dis¬ 
tribution of the transmitted pump beam, in the near field 
(left) and the far-field (right). We observe for these pa- 
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FIG. 2. Saturation of the instability. We show the evolution 
of the patterns as the pump intensity is increased : a) I = 0.24 
W/cm^, (b) I = 0.47 W/cm^, (c) 7 = 1.41 W/cm^ and (d) 
I = 4.24 W/cm^. The detuning is 5 = -1-6.5 F. 


rameters contrasted patterns with a clear hexagonal sym¬ 
metry. However, here the near-held patterns are always 
divided into several domains with different orientations 
of the hexagons, and we never observe the long-range or¬ 
der typically associated with the optomechanical nonlin¬ 
earity [ 2 ^. Using parameters different from the optimal 
set increases the number of such domains, rendering the 
hexagonal symmetry less obvious (see Fig. [5]). 

A key feature of the observed instability is the disap¬ 
pearance of the patterns for large pump intensity (typi¬ 
cally > 2 W/cm^). This behavior is illustrated in Fig. [2] 
Just above threshold (a), the patterns appear in a re¬ 
stricted area around beam center. When the pump power 
is increased, the patterns gain in contrast and spatial ex¬ 
tent (b). A further increase of the intensity leads to a 
progressive blurring of the patterns inside an area around 
beam center, (c) and (d). Note that heating effects due 
to the increase of pump intensity are negligible because 
of the short duration of the pump pulse. This pattern 
blurring is qualitatively different from what is observed 
e.g. for the polarization instabilities in hot vapors where 
no saturation is observed [ 2 ^. In our situation, this sat¬ 
uration is intrinsic to the 2-level description of the atom- 
light interaction as can be seen in Eq. [T] where ri —> 1 as 
s —>■ 00 . 

We have investigated the range of parameters where 
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FIG. 3. (Color on line) (a) 2-level instability domain (<5 > 0). 
We measure Pd (see text) as a function of 5 and /, and plot 
the data as isolines. Note the logarithmic horizontal scale. 
The dots indicate the theoretical threshold for the instability 
(see text), (b) and (c) show cuts through the 2D chart of (a) 
as indicated by the dashed lines. 


the instability can be observed. The result of this study 
is summarized in Fig.[3Ka), where we plot the “diffracted 
power” Pd as a function of pump detuning {S > 0) and 
intensity. Pd is obtained through the following proce¬ 
dure. We first record 30 successive near-held images of 
the patterns like that shown in Fig.jljb), to collect a rep¬ 
resentative sample of shot-to-shot pattern huctuations. 
We then select an area around beam center (of diame¬ 
ter w/2) and perform a 2D numerical Fourier Transform 
(FT). The FT images are then summed, and we extract 
from the resulting averaged FT image the power in the 
pattern mode. This quantity is normalized to the power 
inside the undiffracted beam (central peak in the FT im¬ 
age). We then perform the same operation on images of 
the pump beam, obtained without atoms. This yields the 
background power in the pattern mode, due to the resid¬ 
ual rugosity in the pump’s intensity prohle, which is sub¬ 
tracted from the data obtained in the presence of atoms. 
In addition, in Fig. Pd values are scaled such that the 
maximal value (obtained for 6 = 6.5 F and I = 0.47 
W/cm^) is I. 

On the blue side of the transition, we observe the pat¬ 


terns between roughly i5 = 3.5 F and 17 F (Fig. [3Kb)). For 
smaller values of 6, the patterns vanish quite abruptly. In 
this small-5 regime, the cloud is optically-thick with two 
important consequences: first, the strong absorption con¬ 
siderably reduces the magnitude of the feedback; second, 
there is a large amount of scattered light with a quite 
homogeneous spatial distribution, which is expected to 
blur the transverse field modulation responsible for the 
instability. For large detunings the patterns also disap¬ 
pear but much more gradually, because of the decrease 
oc 1/5 of the refractive index. We observe a well-defined 
lower-intensity threshold for the instability, around 0.16 
W/cm^ for 5 = 6.5 F (Fig. |2Kc)). This threshold is sub¬ 
stantially higher than observed for longer pump pulses, 
where the optomechanical mechanisms sets in 23|. Also, 
we found that a minimum OD of around 100 is required to 
observe the 2-level patterns, while this threshold can be 
considerably lower for optomechanical patterns . The 
saturation of the nonlinearity results in a gradual van¬ 
ishing of the instability for large pump intensity (/ > 2 
W/cm^ for 5 = 6.5 F, which corresponds to s ~ 7). 

We compare in Fig. IHa) our experimental data with 
the theoretical instability threshold (dots) as obtained 
using a 2-level, thin-medium model. This model is based 
on the approach of [2^ , extended to the case of the feed¬ 
back mirror configuration and with the inclusion of ab¬ 
sorption. We also included in our model the longitudinal 
intensity modulation due the interference of the incident 
and retro-reflected pumps, but without the approxima¬ 
tion used in [^. As can be seen, the qualitative and 
quantitative agreement is rather satisfactory. We specu¬ 
late that the discrepancy at small 5 may come from the 
scattered light (not included in the model), as discussed 
above. 

On the red side of the transition (5 < 0), we only 
observe poorly-contrasted structures without clear sym¬ 
metry. The characteristic spatial scale of these structures 
is roughly twice that on the blue side. Their domain of 
observation in (5, I) space approximately mirrors that of 
the patterns on the blue side. A full theoretical expla¬ 
nation for this red-blue asymmetry is still lacking. We 
believe that its origin lies in nonlinear propagation effects 
taking place inside the cloud. This may not come as a 
surprise since such effects have been observed in the past 
in such large cold atom clouds [s^- In that work, we in¬ 
vestigated the self-trapping of a Gaussian beam of small 
waist (20 /im) for 5 > 0, which resulted in a roughly con¬ 
stant transverse size of the beam as it propagated inside 
the cloud. It is thus reasonable to speculate that if an ar¬ 
ray of bright spots such as seen in the transverse intensity 
distribution of Fig. [Ub) forms inside the medium, it will 
be stabilized by self-focusing for 5 > 0. On the contrary, 
for 5 < 0 self-defocusing will tend to blur these struc¬ 
tures. For these effects to play a role, one requires the 
Rayleigh length corresponding to the transverse size of 
the bright intensity spots to be smaller than the length 
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of the medium. This condition imposes a size for the 
spots of a few tens of microns, which is what we typi¬ 
cally observe. 

The Talbot effect and the associated periodic passage 
between phase and intensity modulation is at the 
heart of the transverse instability discussed in this pa¬ 
per. Since we operate well detuned from resonance, a 
transverse intensity pattern mainly induces a transverse 
phase modulation. Propagation to the mirror and back 
can convert this into transverse intensity modulation of 
the backward field, hence phase-modulating the forward 
field, and so on. For a mirror distance d, the transverse 
pattern wavelength A for which this is optimum obeys 
= Xd/{N + 1/4) in the case of a thin medium and of a 
self-focusing nonlinearity In this expression, N 

is an integer of same sign as d. Furthermore, the Talbot 
effect implies that instability thresholds are periodic in 
mirror distance d with period A^/A. 

This tunability and d-periodicity of the pattern scale, 
features specific to the single-mirror feedback scheme, are 
illustrated with Fig. |4] where we plot A (measured by far- 
held imaging of the transmitted pump, see Fig.jljb)) ver¬ 
sus d. The dots correspond to the lowest-q mode, where 
q = 27r/A. The circles correspond to the next higher- 
q mode, which is observed only for large |d|. The bold 
lines are predictions of a thick-medium model, similar 
to that of Fig.lHKa) but including the propagation inside 
the cloud and neglecting absorption. The Talbot period¬ 
icity appears through the fact that the same q, and hence 
A, is observed for periodically-spaced values of d, the d- 
period being h?/\. This is verihed in the insert, where 
the experimentally observed d-period is plotted against 
A (dots) and compared to the expression above (line). 
The overall agreement between experiment and theory is 
very satisfactory, for all instability branches, confirming 
the validity of the Talbot picture in our situation. 

We demonstrated in this paper the existence of a 
pattern-forming optical instability in a cloud of cold 
atoms, based only on the 2-level electronic nonlinearity. 
In this paradigmatic situation, we were able to observe 
the disappearance of the instability at high optical power, 
due to the saturation of the nonlinearity. This work 
demonstrates the interest of cold atomic samples for the 
field of nonlinear optics and pattern formation, motivated 
by the fact that several nonlinear mechanisms coexist and 
can be selected and studied independently. Understand¬ 
ing and controlling these various mechanisms constitutes 
an important step in the future prospect of extending 
these experiments to degenerate quantum gases, where 
the simultaneous self-organization of light and matter can 
lead to a rich class of physical phenomena [s^. 
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UNS, and Region PACA. We acknowledge fruitful discus- 



feedback distance d (mm) 

FIG. 4. Talbot effect and pattern size (d > 0). We mea¬ 
sure A (see text) as the feedback mirror distance d is varied 
(dots: lowest g-mode, circles: higher g-mode). The line is the 
prediction of a thick-medium model. The insert shows the 
measured d-period as a function of the pattern size (stars), 
together with the Talbot effect prediction (line). 
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